System for remote guidance by expert for imaging device

ABSTRACT

Systems and methods for combining tomographic images with human vision. The systems preferably include a first assembly located proximal to an object to be imaged and a second assembly remote from the target. The first assembly is preferably able to collect a tomographic image from a target object and superimpose that tomographic image onto a direct view of the target object. The first assembly also includes components that allow the transmission of the tomographic image to the remote assembly. At the remote location, a highly trained expert may interact with the captured image by placing electronic markers on the image via an electronic interface. The captured image plus electronic marker are then preferably transmitted back to the local site via a second transmitter-receiver. The local user may use the electronic marker to guide his actions during the appropriate task. The systems have particular utility in the medical field where a trained clinician at a remote location may provide guidance to an untrained individual in performing a medical procedure on a patient.

RELATED APPLICATION

This application claims the benefit of U.S. provisional application No.60/893,245 filed Mar. 6, 2007 and is a continuation-in-part of U.S.application Ser. No. 10/126,453 filed Apr. 19, 2002 (now U.S. Pat. No.7,559,895, issued Jul. 14, 2009), which is a continuation-in-part ofU.S. application Ser. No. 09/686,677 filed Oct. 11, 2000 (now U.S. Pat.No. 6,599,247, issued Jul. 29, 2003).

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to image display devices andmethods of using the same. More particularly, the present inventionrelates to methods and devices for combining a reflection of atomographic image with human vision during subcutaneous medicalprocedures in which a user may remotely interact with and provideguidance for the medical procedure being performed.

2. Description of the Background

Because human vision depends at least partially on the detection ofreflected visible light, humans cannot “see” into objects through whichlight does not pass. In other words, humans cannot see into the interiorsections of a non-transparent, solid object. Quite often, and in manydifferent technology areas, this sight limitation may impede or hinderthe effective completion of a particular task. Various partial solutionsto this problem have been utilized in the past (miniature cameras, x-raymethodologies, etc.). However, there is a continued need for improvementto the methods by which the interior of an object is displayed,especially using a real-time imaging modality.

Perhaps in no other field is this sight limitation more of a hindrancethan in the medical field. Clinical medicine often calls for invasiveprocedures that commence at the patient's skin and proceed inward tosignificant depths within the body. For example, biopsy needlesintroduced through the abdominal wall to take samples of liver tissuefor diagnosis of cancer must pass through many centimeters ofintervening tissue. One potential problem with such procedures is thelack of real-time visual feedback in the vicinity of critical structuressuch as the hepatic arteries.

Standard imaging modalities such as Computerized Tomography (CT) andMagnetic Resonance Imaging (MRI) can provide data for stereotacticregistration of biopsy needles within targets in the liver, lungs, orelsewhere, but these methods are typically characterized by the physicaldisplacement of the patient between the time of image acquisition andthe invasive procedure. Real-time imaging modalities offer moreimmediate feedback. Among such real-time modalities, ultrasound may bewell-suited for guidance of needles because it preferably is relativelyportable, is inexpensive, produces no ionizing radiation, and displays atomographical slice, as opposed to angiography, which displays aprojection. Compared with angiography, ultrasound may offer theadditional advantage that clinicians are not rushed through proceduresby a desire to keep exposure times to a minimum.

Conventional two dimensional (2D) ultrasound is routinely used to guideliver biopsies, with the needle held in a “guide” attached to atransducer. The guide keeps the biopsy needle in the plane of the imagewhile the tip of the needle is directed to targets within that sameplane. This system typically requires a clinician to look away from hishands at a video monitor, resulting in a loss of direct hand-eyecoordination. Although the clinician can learn this less direct form ofcoordination, the natural instinct and experience of seeing one's handsbefore one's eyes is preferred.

As a further disadvantage, the needle-guide system constrains the biopsyneedle to lie in the image plane, whereas the clinician may prefer theneedle to intersect the image plane during some invasive procedures. Forexample, when inserting an intravenous (IV) catheter into an artery, theoptimal configuration may be to use the ultrasound image to visualizethe artery in cross-section while inserting the needle roughlyperpendicular to the image into the lumen of the artery. The prior artsystem just described may not be capable of accomplishing this task.

A related visualization technology has been developed where threedimensional (3D) graphical renderings of previously obtained CT data aremerged with an observer's view of the patient using a partial orsemi-transparent mirror, also known as a “half-silvered” mirror. Apartial mirror is characterized by a surface that is capable of bothreflecting some incident light as well as allowing some light to passthrough the mirror. Through the use of a partial mirror (or otherpartially reflective surface) a viewer may see an object behind thepartial mirror at the same time that the viewer sees the image of asecond object reflected on the surface of the mirror.

The partial mirror-based CT “Image Overlay” system requires independentdetermination of location for both patient and observer using external6-degree-of-freedom tracking devices, so as to allow appropriate imagesto be rendered from pre-acquired CT data.

Another recently developed imaging technology merges ultrasound imagesand human vision by means of a Head-Mounted Display (HMD) worn by thehuman operator. The location and orientation of the HMD is continuouslydetermined relative to an ultrasound transducer, using6-degree-of-freedom tracking devices, and appropriate perspectives ofthe ultrasound images generated for the HMD using a graphics computer.

These prior art systems may not be appropriate for use with a practicalreal-time imaging device. Controlling the multiple degrees of freedomcan be difficult, and the systems may have too many complex parts to beuseful. As such, there is recognized a need in the art to provide adevice capable of merging a human's normal vision of an object with an“internal” image of the object that emphasizes freedom of operatormovement and/or simplicity of design.

Numerous circumstances arise where the expertise of a trained specialistmay be advantageous or required for the performance of a procedure. Forexample, a medic in the field of battle may be required to access thefemoral vein of a wounded soldier. While the medic may take advantage oflocal tomographic imaging devices, the expertise of a medicalprofessional would prove invaluable in performing the procedure.However, it may be physically impossible for the medical professional tobe present in sufficient time to save the life of the solider. There hasbeen a long-standing need in the medical imaging community for systemsand methods that allow a remotely located expert to take advantage oflocally generated tomographic images to aid in performing a task. Thepresent invention addresses that need.

SUMMARY OF THE INVENTION

The present invention contemplates, in at least one preferredembodiment, a device and method for merging human vision of the outsideof a target object and a reflected tomographic image of the internalfeatures of the same object. The invention may include an image capturedevice (e.g., a tomographic scanning device such as an ultrasoundtransducer), an image display device (e.g., a computer or videomonitor), and a half-silvered mirror to “fuse” or superimpose the twoimages together.

In at least one preferred embodiment, the present invention provides a2D ultrasound transducer, an image display, and a partially reflective,partially transparent, surface (e.g., half-silvered mirror) generallydisplaced between a target object and the image display. The transducer,the display, and the mirror may be fixedly attached to each other, orone or more elements may be partially or completely moveable withrespect to the others. The movement may be accomplished through directmanipulation by the operator or with the use of one or more roboticarms.

In at least one preferred embodiment, the present invention provides a3D ultrasound transducer, an image display, and a partially reflectivesurface broadly displaced between a target object and the image display.The image display may preferably display an appropriate slice of the 3Dultrasound data (effectively a 2D tomographic image) to enable a propercombined image to be seen when an observer looks at the target objectthrough the partially reflective surface.

In at least one preferred embodiment, the present invention includes aseries of gears, pulleys, or other motion transfer devices installedbetween the transducer, the display and the half-silvered mirror toallow the angle between the mirror and display to follow the anglebetween the transducer and the mirror as the transducer is moved. Thepresent invention also contemplates various embodiments where thetransducer is free to move in any direction or where robotic systemsallow for the remote performance of procedures.

In other preferred embodiments, the present invention encompassessystems and methods where a local device capable of merging a directview of a target with a tomographic image of the target is adapted totransmit that information to a remote site. The images that aretransmitted remotely may include only the tomographic image or both thetomographic image as well as the direct view of the target. At theremote site, the present invention preferably includes a display screenor other method of display at which a remote user may view thetransmitted images. Through that display, the remote user may provideinterpretation or guidance to the local user. The remote user may alsoplace an electronic cursor or other marker onto the images to inform alocal user of an appropriate location for a procedure to be performed.The cursor location may then preferably transmitted to and displayed onthe local user's screen and reflected to the appropriate location withinthe patient to provide guidance, interpretation, or instruction to thelocal user.

These and other details, objects, and advantages of the presentinvention will be more readily apparent from the following descriptionof the presently preferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

For the present invention to be clearly understood and readilypracticed, the present invention will be described in conjunction withthe following figures, wherein like reference characters designate thesame or similar elements, which figures are incorporated into andconstitute a part of the specification, wherein:

FIG. 1 is a schematic view of a device capable of merging a reflectedtomographic image from a 2D ultrasound scanner with a direct view of atarget image;

FIG. 2 is a schematic of the image angles that allow the operator tomove in relation to the half-silvered mirror while maintaining imagemerger;

FIG. 3 is a schematic view of a device capable of merging a slice from a3D scan with a direct view of a target image;

FIG. 4 is a schematic view of an imaging system with the image capturedevice removed from the remainder of the system;

FIG. 5 shows the methodology applied to remote operation using a mockeffector in the field of view;

FIG. 6 shows the methodology applied to remote operation utilizing aremote-controlled effector;

FIG. 7 is a schematic diagram of the present methodology applied to thefront of a large imaging machine such as a CT scanner;

FIG. 8 shows an image acquisition device inserted into a jig (8A) andthe jig with the image acquisition device replaced with a screen/mirrorapparatus (8B);

FIG. 9 shows a calibration system for the present invention (9A) withexploded views of tubes attached to the target before (9B) and after(9C) gel is added;

FIG. 10 shows a light source guidance method according to the presentinvention;

FIG. 11 details a reflective surface embedded between two surfaces withequivalent refractive properties;

FIG. 12 shows an embodiment of the present invention with a holographicplate;

FIG. 13 shows exemplary material layers combined in an LCD;

FIG. 14 shows an embodiment of the present invention utilizinginterference principles; and

FIG. 15 displays an embodiment of the present invention where thetomographic imaging portion of the present invention is adapted totransmit images to a remote location;

FIG. 16 shows an embodiment of the present invention where the imaginginformation is received remotely and may be observed by a remote user;and

FIG. 17 shows an embodiment of the present invention where a digitalcamera is used to transmit information to a remote location.

DETAILED DESCRIPTION OF THE INVENTION

It is to be understood that the figures and descriptions of the presentinvention have been simplified to illustrate elements that are relevantfor a clear understanding of the invention, while eliminating, forpurposes of clarity, other elements that may be well known. The detaileddescription will be provided hereinbelow with reference to the attacheddrawings.

The invention contemplates, in at least one presently preferredembodiment, a method and device for merging or superimposing thereflection of a two dimensional tomographic image of the interior of atarget object with the normal human vision view of the outside of thatsame target object. This methodology may be used in any applicationwhere viewing the interior of an object is desired, and the methodologyis not limited to any particular industry or application. In someembodiments, the present invention is applied to medical procedureswhere an in situ image of tissue provides valuable information for themedical practitioner. In other embodiments, the present invention may beused to image the interior of an industrial device and provide valuableinformation about the relative positioning of parts within the devices.In still other embodiments, the present invention may be used to imagethe interior of a mine, artillery shell, or other munition whosedeactivation is desired. The interior image is preferably captured byany real-time imaging modality, where real-time does not necessarilyindicate near-instantaneous display, but only that the target object hasnot moved significantly since the scanning was performed. Examples ofreal-time imaging modalities include ultrasound, radar, LIDAR, andSONAR.

Although this methodology and device can be used across many differentfields of endeavor, the present invention may find particularapplicability in the medical field. Because unwarranted or excessiveintrusion into the interior portions of a human body may cause damage,infection, or other unwanted effects, these intrusions should be limitedin both the number of instances and the scope of the intrusion. As such,it is preferable to perform subcutaneous procedures with at least somedirect sighting of the interior of the patient. Because the medicaldevice applications may be particularly useful, the present inventionwill be described with reference to such a medical device, but thisexemplary disclosure should not limit the scope of this patent to anyparticular industry or use.

FIG. 1 shows an isometric view of a presently preferred embodiment of animaging device 10 utilizing a two dimensional (2D) ultrasound transducer12 capable of taking a B-Mode image slice 14 of a target object 16. InFIG. 1, there is a two dimensional ultrasound transducer 12 fixedlyattached to a rigid frame 18. This transducer 12 is preferably aconventional ultrasound transducer that captures a “sonic” tomographicimage slice 14 of the interior portion of the target object 16 (in thiscase a human patient).

Extending vertically from the middle region of the rigid frame 18 is ahalf-silvered mirror or other semi-transparent, semi-reflective,material 20. The half-silvered mirror 20 allows a user 22 (e.g., adoctor) to look through the mirror 20 at a target object 16 (e.g., apatient) located on the other side of the mirror 20 at the same timethat a second image 24 is reflected on the front surface of the mirror(at 28). In this way, the direct target object image 26 and thereflected tomographic image 24 can be combined (image line 30) in thefield of view of the user 22.

In FIG. 1, the half-silvered mirror 20 is depicted extending verticallyup from the ultrasound transducer 12 midway along the transducer handle,but, in fact, the mirror 20 may be located at some other position inanother vertical plane either behind or in front of the depictedvertical plane. More specifically, the FIG. 1 half-silvered mirror 20could be translated forward or backward (or even tilted) as long as thedisplay 32 is moved in a way that corresponds appropriately (asdescribed in detail below).

At the opposite end of the rigid frame 18 from the transducer 12 is aflat panel display 32 showing the ultrasound image or other tomographicslice 34, with the image portion 34 facing upwards. This display 32 maybe any low profile or flat display and may preferably be a liquidcrystal display (LCD). When a user 22 looks at a target object 16through the half-silvered mirror 20, the ultrasound display image 34will be reflected along line 24 onto the front face of the half-silveredmirror (at 28). The user's sight line 30 will therefore be a combinationor superimposition of the direct target object image 26 and thereflection of the ultrasound image 24.

In order to correctly visually merge the reflected ultrasound image 24with the target object image 26, the ultrasound display image 34 may bereversed (along a horizontal plane), flipped (along a vertical plane),rotated, translated, and/or scaled (depending on the original image 34location, orientation, and scale on the display 32) so that thereflected ultrasound image 24 on the face of the half-silvered mirror(at 28) correctly portrays the size, scale, and orientation of theultrasound slice 14 being taken. In a practical sense, if one merelyrotates the transducer 12 180 degrees, the ultrasound display image 34will be flipped exactly as if this image manipulation was accomplishedelectronically.

A profile of a human operator's eye 22 is shown in FIG. 1 lookingthrough the half-silvered mirror 20 at the target object 16 (patient).Because of well-known laws of light reflection, the ultrasound image 34on the flat-panel display 32 will be reflected on the operator-sidesurface of the half-silvered mirror (at 28). Therefore, as the operator22 looks at the target object 16 through the half-silvered mirror 20,the reflected ultrasound image 24 is merged (superimposed) with or ontothe direct target object image 26. To the operator 22, these two images24, 26 will effectively combine into one image 30 that includes thesurface (normal vision 26) of the target object 16 and the interior(reflected ultrasound 24 or other tomographic reflection) of the targetobject 16. Because the angle of reflection of the ultrasound imagefollows the operator's sight angle as the operator's head moves, themerger 30 of these two images 24, 26 is independent of the location ofthe operator 22 (user). Therefore, the user 22 can move his head as wellas take full advantage of stereoscopic vision to extrapolate the hiddenparts of the invasive tool (e.g., needle) from the exposed parts of thesame tool with respect to the anatomical structures in the ultrasoundscan.

Because the direct target object image 26 and the reflected ultrasoundimage 24 are combined or superimposed on the surface of a half-silveredmirror (at 28) that may be naturally within the operator's direct lineof sight (along 30), the operator 22 can preferably maintain directhand-eye coordination throughout the procedure. This naturalline-of-sight image combination 30 effectively allows the operator 22 tosee “through” the surface (e.g., skin) of the target object 16 and intothe underlying structures or layers of materials.

Further, although the present imaging device 10 may be used withvirtually any imaging technology, using a nearly instantaneous imagingtechnology, such as ultrasound allows the interior and exterior views ofthe target object 16 to be nearly synchronous. However, the method maybe applied to any real-time tomographic imaging modality, where“real-time” refers to any imaging modality that can update the displayedimage 34 before the patient (target object 16) moves. As patientmovement decreases, “slower” modalities become more useful. If a slowerimaging technology is used (i.e., there is a substantial lag timebetween image capture and image display), the operator 22 may instructthe patient 16 to lie still so that the delayed interior image 34 willremain aligned with the current target object image 26. In this way,even a slower imaging technology may be used with the present invention.

Some possible “quick” imaging modalities include ultrasound, cine-CT andrapid MRI. Some “slower” modalities include conventional MRI,conventional CT, SPECT and PET. However, even these slow modalities maycreate an accurate combined image 30 so long as the target object 16 hasnot moved since the last image was captured. A needle or other intrudingdevice may still be introduced using the overlaid image for guidance,provided the target object 16 has not moved.

To increase the likelihood that the patient remains still, somecombination of laser or ultrasonic range-finders, video cameras, and/ormotion sensors (not shown) may be used to detect such movement and warnthe operator 22 that the image will not be superimposed perfectly (at28). Alternatively, these same sensor devices could detect exactly howfar the target object 16 has moved since the last image capture andelectronically correct the displayed image 34 and/or the location of thedisplay 32 or mirror 20 (see below) to compensate for such target objectmovement.

The mathematical requirements for locating the components of theapparatus are shown in FIG. 2. The half-silvered mirror 20 is preferablypositioned between the tomographic slice 14 and its image 34 reflectedon the flat-panel display, separated from each by the same angle θ(θ₁=θ₂=θ). In essence, the half-silvered mirror 20 bisects the angle 2θ.As seen in FIG. 3 (below), θ can approach zero. Because the mirror 20 inFIG. 2 bisects the angle 2θ, point P in the ultrasound slice 14 and itscorresponding image P′ in the flat panel display 34 are both distance dfrom the half-silvered mirror 20. The line between the point P in theslice 14 and its image P′ in the display 34, along which d is measured,is orthogonal to the plane of the semi-transparent mirror 20.

The figure shows the eye of the viewer 22, to whom the ultrasounddisplay will be superimposed (along 30) on the corresponding physicallocation of the slice irrespective of the viewer's location. The angleof incidence from the flat panel display 34 to the face of thehalf-silvered mirror (at 28) is labeled α₁ in FIG. 2. By well-known lawsof light reflection, the angle of reflection α₃ is equal to the angle ofincidence α₁. Because the mirror 20 bisects 2θ and further by well-knownlaws of geometry, the “incidence” angle α₃ from the corresponding point14 in the target object 16 to the back of the half-silvered mirror 20 isalso equal (α₁=α₂=α₃=α). In this way, regardless of viewer position, thedirect target object image 26 and the reflected tomographic slice image24 will always coincide to combine image 30.

FIG. 3 shows a presently preferred embodiment of the imaging device 10utilizing a three dimensional (3D) ultrasound transducer 12. As with the2D transducer described above, the FIG. 3 embodiment details ahalf-silvered or partial mirror 20 fixedly attached to the transducer 12and extending vertically upwards therefrom. In this embodiment, thetransducer 12 is capable of capturing 3D imaging data of a scannedvolume 54 (e.g., a Real Time 3D (RT3D) ultrasound image). The image 34that is shown on the flat-panel display 32 (and therefore reflected 24onto the partial mirror 20) is preferably a 2D tomographic slice throughthe scanned volume 54 in the target object 16 (for example, a “C-Mode”slice, parallel to the face of the transducer 12). This 2D tomographicimage 34 may be mathematically computed from the collected 3D imagingdata by a computer (not shown). The flat-panel display 32 should beproperly located and oriented to precisely reflect 24 onto thehalf-silvered mirror (at 28) the location of the correspondingtomographic image within the target object 16. Once again, the image 34on the display 32 is preferably electronically translated, rotated,scaled and/or flipped to complete proper registration independent ofviewer location, as necessary.

Compared to the “Image Overlay” CT-based system using previouslyobtained data or any other “lagging” imaging scheme, ultrasound or other“real-time” data is preferred so that the present location of thepatient (target object) 16 need not be independently established orregistered by the imaging device. Whatever is currently in front of thetransducer will simply appear superimposed on the operator's visualfield at the appropriate location. Furthermore, the present inventionpreferably displays only a single slice, as opposed to a complete 3Drendering as in the “image overlay” CT system (described above).Therefore, the visual image merger 30 can be made independent of theobserver's location simply by placing the ultrasound display 32 whereits reflection 24 in the half-silvered mirror 20 superimposes on thedirect view 26 of the target object 16. Since the displayed tomographicimage 34 is 2D and is reflected precisely on its proper location in thetarget object 16, the correct combination 30 of these views 24, 26 isindependent of viewer 22 location. This may be simpler and moreefficient than superimposing 3D renderings.

The devices and methods as described above include rigidly attaching asemi-transparent mirror 20 and flat-panel display 32 to the tomographicscanning device 12 (or other image capture device). This rigid fixationand the associated bulk of the complete device 10 may reduce the abilityof the operator 22 to manipulate the scanning device or transducer 12during a procedure. There are several ways in which the freedom andability of an operator 22 to manipulate the image capture device 12 maybe increased.

For example, a linkage system of levers, weights, pulleys, and/orsprings (not shown) could be constructed, while maintaining the rigidrelationship between the device components (scanner 12, mirror 20,display 32), to aid in the manipulation of the entire apparatus 10. Thislinkage system of levers, weights, pulleys, and/or springs maycantilever or otherwise reduce the amount of force necessary tomanipulate the apparatus 10. A similar configuration, for example, isoften used in hospitals to aid in the manipulation of heavy lightsduring surgery. These levers, weights, pulleys, and/or springs arepreferably attached to the ceiling or to a floor stand.

Alternatively, the operator 22 may obtain greater flexibility tomanipulate the transducer 12 through a system of gears and/or actuatorsthat keep the angle (θ₁) between the display 32 and the half-silveredmirror 20 equal to the angle (θ₂) between the transducer 12 and themirror 20. As the user 22 moves the transducer 12 in various ways,additional gears, actuators, and/or encoders of linear and/or angularmotion could accommodate this transducer motion (and correspondingchange in θ₂) by providing equivalent motion of the displayed ultrasoundimage 34 (and corresponding change in θ₁) through physical manipulationof the display screen 32 and/or electronic manipulation of the image 34displayed on the screen 32. These gears, actuators, and/or imagemanipulations preferably keep the appropriate angle (θ) and locationbetween the display 32 and the half-silvered mirror 20 so that the user22 can move the transducer 12 and still see a proper combination image30 of the target object image 26 and the reflected ultrasound image 24independent of viewer location. Such a system could be made toaccommodate 6 degrees of freedom for the transducer, including 3rotations and 3 translations. As with the above embodiments, thisembodiment preferably entails physical attachment of the transducer 12to the rest of the apparatus (which may hinder use of the device 10).

To further increase the ability of the operator 22 to manipulate thetransducer 12 (or other tomographic scanning device), the transducer 12(or other image acquisition device) may be physically freed from therest of the apparatus. By continuously determining the relative anglesand location of the transducer 12 with respect to the half-silveredmirror 20 using a system such as the commercially available“FLOCK-OF-BIRDS” or “OPTITRACKER” systems, the angle (θ₁) andorientation of the display 32 with respect to the mirror 20 couldlikewise be adjusted to compensate for this transducer manipulation. Oneembodiment of a system for freeing a 3D ultrasound transducer 12 isshown in FIG. 4. The appropriate slice through the 3D ultrasound data 54is computed and displayed (on display 32) so as to effect a merger 30 ofthe two images 24, 26 on the face of the mirror 20.

Similarly, for 2D ultrasound, the manipulability of the transducer 12may be especially important when searching for a target. The problem canpreferably be addressed by detaching the transducer 12 from the rest ofthe assembly (the mirror 20 and the flat-panel display 32). A 6-degreeof freedom tracking device such as the “FLOCK-OF-BIRDS” or“OPTITRACKING” system may be attached to the handle of the transducer12. The flat-panel display 32 may be detached from the mirror 20 andcontrolled by a series of motors such that the display 32 would be madeto remain exactly in the reflected plane of the ultrasound slice, asdetermined by the tracking system on the transducer handle. Such displaymovement will preferably occur according to well-known principles ofrobotics.

Such a motorized device may lag behind the movement of the transducer 12during rapid manipulations of the transducer 12 by the operator 22, butwould preferably catch up with the operator at relatively motionlessperiods when the operator 22 had located a desired target. The mirror 20may preferably be held motionless relative to the target object 16,establishing the frame of reference for both the transducer trackingsystem and the motorized display 32. Alternatively, the mirror 20 may bemotorized and the display 32 held constant (or both the mirror and thedisplay could move).

The other degrees of freedom which may be necessary to visually fuse 30(superimpose) the displayed ultrasound image 24 with the actual targetimage 26 may be supplied by graphical manipulation of the displayedimage 34 on the flat-panel display 32, based on the tracking of thetransducer 12. As with the fixed and geared assemblies described above,the motorized display 32 and graphical manipulation of the displayedimage 34 preferably provides visual “fusion” 30 of the reflectedultrasound image 24 with the actual target object image 26 independentof operator 22 or target object 16 location.

In one presently preferred embodiment of the invention, two roboticarms, or a single paired robotic device, manipulate both the transducer12 and the display 32 (and/or the mirror 20) under remote control insuch a way that the visual fusion 30 is maintained. This may eliminatethe need to track the transducer 12, replacing it with feed-forwardremote control of the transducer location via a joystick or othercontroller. The simultaneous control of two robotic devices whosemotions may be as simply related as being mirror images of each other,may be accomplished in a fairly straightforward manner, and may exhibita more synchronous image fusion.

A natural pivot-point for the display monitor may be the reflection ofthe point of contact between the transducer and the target objectbecause, during many procedures, the operator tends to rotate theultrasound transducer in all three rotational degrees of freedom aroundthis point (to find a desired target). Thus, for the simultaneouscontrol of two robotic devices just described, rotating the displaymonitor with three degrees of freedom around this point may bepreferred. For systems that move the display while tracking a manuallymanipulated transducer, at least one translational degree of freedom maybe needed to allow the display monitor to become coplanar with theultrasound slice.

Calibration of the fixed system and development of the servo-linked(motorized) display system or the dual robotic system just described mayrequire careful consideration of the degrees of freedom in theregistration process. First, consider only the geometric transformation,i.e., assume the scale of the captured slice and the displayed image areidentical and undistorted. To complete the geometric transformregistering the reflection of the ultrasound image to the actual slice,we need to satisfy 6 degrees of freedom. First we have 3 degrees offreedom to manipulate the display physically into the plane of the slicereflection. This can take the form of two rotations to make the displayscreen reflection parallel to the slice and one translation orthogonalto the display screen to bring it into precisely the same plane.

Once the display reflection and the slice are in the same plane, we need3 more degrees of freedom to match the image and the slice, which may beachieved through two translations and one rotation of the image on thedisplay. In essence, the 6 degrees of freedom place the display in theproper physical plane to reflect the image on the half-silvered mirror(3 degrees of freedom) and then rotate and translate the image on thedisplay so that the correctly placed reflection is properly aligned (3additional degrees of freedom) on the mirror with the actual targetobject image.

Beyond the geometric transformation, further calibration may berequired. First, the proper scale must be calibrated. This includesisotropic scale (similarity transform) and non-isotropic scale (affinetransform). Further corrections may be required for non-linear geometryin both the imaging system and the display by warping of the image.

To the extent that the geometric properties of the slice do not changewith tissue type, and the slice geometry does not change as thetransducer is moved relative to the target object, calibration of thesystem 10 may only need to be performed initially, using a phantomtarget object (not an actual patient). Such calibration will suffice forslice geometry due only to the scanner. Further changes in imagegeometry due to tissue properties will depend on transducer locationrelative to the tissue. These changes may be due to differences in thespeed of sound in different tissue types. It may be possible to correctfor these using image analysis techniques as known and developed in theart.

A problem with calibration may arise because a phantom in a water tankthat is easily scanned using ultrasound will appear displaced to humanvision due to refraction at the air-water interface. Several solutionsare described here to this problem. One solution may use a rod thatintersects both the reflected image (in air) and an ultrasound slicedisplaced along the rod (water). The display may then be physicallymoved or rotated, and the image on the display may be electronicallymoved or rotated, to make the rod appear to intersect the correspondingreflected ultrasound image appropriately.

A second calibration solution includes the use of a calibration phantom.The phantom is placed in water or some other ultrasound transmitting(but light refracting) medium and scanned by the image capture device.The image is “frozen” (still picture) on the display and reflected offof the half-silvered mirror. Without moving the calibration phantom, thewater or other medium is drained or removed from the calibration setup.The user can then adjust the display or the image on the display untilthe “frozen” reflected scan image of the phantom aligns with the directsight image of the calibration phantom. Many other calibration schemescould be used within the scope of the present invention.

In one presently preferred embodiment of the present invention, a“remote” procedure is performed through the use of a tomographicscanning device and surgical implements controlled remotely with amechanical or robotic “mock effector” in the operator's field of viewinstead of the actual target object being in the operator's field ofview. A mock effector is a physical replica of the actual invasiveinstrument, preferably of identical shape but not necessarily identicalscale. The mock effector may either be directly controlled by theoperator (FIG. 5) with mechanical linkages, encoders and/or trackingdevices relaying the desired motion to the actual effector, oralternatively the operator may use a remote control to activate both themock effector and actual (surgical) effector with corresponding motions(FIG. 6).

For example, a procedure may involve using a hypodermic needle ormicro-pippette to take samples in a certain plane. In FIG. 5, thetomographic scanning device (not shown) may be aligned to capture aslice 14 of the target object from which the sample may be taken. Theaction of the actual surgical needle 70 may be controlled remotely byoperator 22 manipulation of a mock effector (needle) 70 in the user'sfield of view demonstrating the precise motion of the actual remoteeffector 72, although possibly at a different scale. The scale (as wellas the location and orientation) of the mock effector 70 would matchthat of the reflected tomographic image 24 (to make the combined image30 an accurate representation).

In this example, a mock needle 70 may be present in front of theoperator 22 (remote from the target object). The operator 22 preferablylooks through the half-silvered mirror 20 at the end of the mockeffector 70. The operator's field of vision is a merged image 30 of thedirect view 26 of the mock effector and the reflection of thetomographic slice 24. If the actual needle 72 performing the procedureon the patient (target object 16, not shown, through which tomographicslice 14 is acquired) is of a different scale than the mock effectorneedle 70, then the tomographic image 34 displayed on the monitor 32 ispreferably moved and/or scaled so that the reflected tomographic image24 and the direct view 26 of the mock effector are of equal size,scaling, and orientation at the surface 28 of the half-silvered mirror20.

The mock effector needle 70 and the actual surgical effector needle 72are preferably connected through some sort of control mechanism 74. InFIG. 5, this control mechanism is shown as a direct mechanical link 74,being a rod fixed at one end by a ball-and-socket 57 to allow 3 degreesof rotation. As the operator manipulates the mock effector 70, theactual effector 72 will move correspondingly (although on a smallerscale). Similarly, the control mechanism may be some type of tracking orencoder device that registers the movement of the mock effector 70 andtransfers this movement to the actual surgical effector 72. In this way,the operator can manipulate a mock effector 70 and cause a procedure tobe performed on a target object at a remote location.

This remote procedure model may be useful for extremely small scaleprocedures. For example, assume a microscopic region of a patient mustbe cut (e.g., a cancer cell removed or a cornea operated upon). In theregion of a very small cutting implement, a specialized tomographicscanning modality (such as Optical Coherence Tomography (OCT) or a highfrequency (100 MHz) ultrasound) may be used to capture an image slice.At a remote location, a doctor may preferably look through a partialmirror with a reflection of the tomographic slice on its facesuperimposed upon a mock cutting implement whose motion is linked tothat of the actual cutting implement. Although the actual procedureoccurs on a microscopic scale, both the tomographic slice and the mockeffector can be magnified or scaled up to a point that allows the doctorto perform the procedure in a more relaxed and accurate manner. As longas the mock effector is scaled up to a similar size as the tomographicslice, the overlay of the images may be accurately located and orientedwith respect to each other. In this way, small scale medical (ornon-medical) procedures may be easier to perform. Similarly, large scaleprocedures such as undersea robotics using sonar-based tomographicimaging may be performed remotely at a smaller scale than they actuallyoccur.

FIG. 6 details one possible robotic version of the present invention.The FIG. 6 remote application is generally similar to the FIG. 5implementation with the addition of a control box 84 used to control themotion of both the mock effector 80 and the actual surgical effector 82.As in the previous example, the operator 22 looks through the surface ofthe half-silvered mirror 20 at the working end of a mock effector 80. Atomographic image 34 of the target object 16 is displayed on a monitor32 and reflected along line 24 onto the surface of the half-silveredmirror 28. The operator's filed of view includes the merger 30 of thesetwo images 24, 26.

In this example, however, the operator 22 preferably does not directlymanipulate the mock effector 80. Instead, some type of control, forexample a joystick, keyboard, voice activated software, or other device,is manipulated by the operator 22. This control device causes themovement of both the mock effector 80 (through control line 86) andactual effector 82 (through control line 88). In FIG. 6, each of theseeffectors 80, 82 can be moved with 3 degree of freedom movement. Themock effector 80 can again be scaled larger or smaller to make themanipulation of the actual effector 82 more convenient. Preferably thesize, scale, and orientation of the tomographic image 34 displayed onthe monitor 34 is matched to the size, shape, and orientation of themock effector 80.

For robotic versions of the present invention, the effector 82 thatinteracts with the patient 16 need not necessarily be a mechanicalsurgical tool. For example, the effector could be a laser, RF (radiofrequency) transmitter, or other device for delivering or impartingenergy or matter on a target object. In these cases, the mock effectorused by the operator may include some kind of demonstration of theenergy or matter delivered, either expected or measured, to the patient.For example, an isosurface of expected RF field strength may bephysically constructed and mounted on the mock effector used by theoperator such that the field model intersects the reflected imageappropriately. In this way, the operator can take into account the fieldof the effector's use, as well as the effector itself.

The present invention may also depend on the lighting used on or aroundthe device. For example, light that hits the surface of thehalf-silvered or partial mirror from above (operator-side) may introduceunwanted reflections in the semi-transparent mirror. In this case, thetarget object will be more difficult to see. Alternatively, light thatcomes from a source beneath the half-silvered mirror (on the same sideof the mirror as the target object) may increase the clarity of thetarget object image without introducing unwanted light reflections ontothe half-silvered mirror. Various types of lighting (visible,ultraviolet) as well as paints, markings, and light emitting markers onthe targets or tools themselves may have different properties that areadjustable to change the contrast, intensity, and interpretability ofthe image superimposition.

Alternative forms of light may also be used to register locations in theultrasound during a procedure. These alternative light sources can beused to identify certain features of the target object in addition tothe 3D visual cues inherent to superimposition of the reflected image.For example, a plane of laser light can be created with a movable mirrorand a laser such that any real object (part of target object or mockeffector) that intersects the plane of laser light will be “marked” bythe colored lines of the laser light. Such a laser marking system couldbe used with a computer vision system to permit automated detection andlocation determination of the intersection point of the located objectand the light plane. This system may be used for automated calibrationwith corresponding features detected in the tomographic image.

Light sources could also be arranged relative to opaque shields so thatonly certain parts of the target object are illuminated, such as allparts beyond (or nearer to) the reflected tomographic image. Thus theimage would fall on what would effectively be a clipping plane throughthe object, with all parts of the image closer to (or further from) theviewer not illuminated. Sound, tactile, and/or other forms of feedbackmay be provided to the operator based on the location of tools relativeto the reflected image. These feedback indicators may alert the operatorwhen contact is made, for example between the tip of a needle and thereflected slice.

Various techniques may be used to alter the image as viewed on thelow-profile display. The image may be rotated, translated, scaled, ordistorted, as previously described, or otherwise cropped or manipulatedin many ways according to the needs of the user of the system. Forexample: extraneous parts of the image may be removed; specificanatomical targets may be automatically identified and graphicallyenhanced; surgical tools may be tracked and their hidden sectionsgraphically simulated; and/or other useful information may besuperimposed on the displayed image for the operator relating to theinvasive procedure.

In all, the present invention may be useful for many medical proceduresincluding amniocentesis; many forms of surgery; soft tissue biopsy oforgans such as liver, kidney, or breast; and procedures such as theinsertion of central venous lines, or even peripheral intravenouscatheters. In brain surgery, for example, deformation of the brain afterremoval of portions of the skull leads to inaccuracy of registration innon real-time modalities such as conventional CT. Real-time guidanceusing ultrasound may compensate for such deformations, as well asprovide adaptive guidance during the removal of an abscess, for example,or in other cases where structures may change shape or location duringprocedures. Another example is monitoring and correction of tissueinfiltration during the infusion of cancer drugs into large veins. Theinvention may positively affect the success and flexibility of these andother invasive procedures.

Since the visual image merger is independent of viewer location, two ormore human operators may work together in the same field of view,assisting each other manually and/or offering consultation. Theinvention may be valuable in teaching, for example, by clarifying thecontent of ultrasound images through its location in the target object.

As briefly mentioned above, the version of the device using a mockeffector could be used at microscopic scales, for example, to insertmicro-pipettes into individual cells under OCT guidance to gatherintracellular samples to determine whether the cells are of a cancerousnature (or to deliver therapy to a single cell). Other possible examplesinclude the use of very high frequency ultrasound to guide microscopicsurgery on the cornea, and high resolution MRI to guide biopsies oftissue samples of small animals. It may also be used to guide theadministration of radiation and other non-invasive procedures, to guideprocedures at the end of a catheter or endoscope equipped with atomographic scanner, or in many other technical and non-technicalapplications.

One or more of the above embodiments may be oriented toward a portableversion of the present invention. The size, shape, and materials usedmay be minimized so that the entire apparatus can be carried by a singleuser (or a few users) to the site of a procedure. An ultrasoundtransducer is preferably used in the portable embodiment because of itssmall size and ease of use. These embodiments may be especially suitedfor use in the battlefield for the removal of foreign bodies such asbullets or shrapnel.

The ultrasound transducer in the above portable version may be replaced(towards the opposite end of the size spectrum) with a comparativelymassive CT or MRI scanner (see, FIG. 7). The principle of operation isstill based on a controlled geometric relationship between the scanner,the mirror 20, and the display 32, just as in the above embodiments. Inessence, the angle (θ₁) between the display 32 and the half-silveredmirror 20 should be equal to the angle (θ₂) between the mirror 20 andthe slice 91 through the target object within the gantry of the CT orMRI scanner 90.

The image from a CT or MRI machine can often be converted into anappropriate tomographic slice within less than one minute from the imagescanning. Once a CT scanner is no longer transmitting X-rays (after theimage is captured), there will be no harmful exposure to the operator.As seen in FIG. 7, the gantry 90 of these machines may provide ampleaccess for a doctor or other user 22 to perform an invasive procedure ona patient within the CT machine. Furthermore, if the space in the gantry90 is not sufficient for a particular procedure, the patient (or othertarget object) may be moved out of the machine a known distance, and theimage 34 of the tomographic scan may then be shifted by that same amount(provided the patient did not move in any other way).

There are many other embodiments and alternatives that may be used incombination with the general structures of the present invention asdescribed above. For example, a jig can be used to “store” the geometricrelationships between the image acquisition device (tomographic scanner)and the target object to allow for the use of the invention without theimage acquisition device being present upon viewing.

FIG. 8A shows one exemplary embodiment of a jig 800 that may be used tostore the geometric relationships of the system. The target object 805and the image acquisition device 810 (in this case an ultrasoundtransducer) are mounted in a jig 800 using one or more pegs 815. The jig800 is used to define the spatial relationships between the ultrasoundtransducer 810 and the target object 805. Once in position, atomographic slice image 820 of the target object 805 can be obtained andstored for later use.

FIG. 8B shows the same jig orientation with the data acquisition deviceremoved. In its place, a display assembly 825 is inserted into the jig800. In much the same way as previously described embodiments, FIG. 8Bshows a half-silvered mirror (partially reflective surface) 830 orientedbetween the viewer's eye and the target object 805. Further, a flatpanel or other display 835 is connected to the jig 800 such that itreflects a captured image onto the user-side face of the half-mirroredsurface 830.

If the image capture device 810 from FIG. 8A is used to capture an image820 of the target object 805, and the flat panel display 835 of FIG. 8Bis used to display the captured image 820 at a later time, the reflectedview of the interior portion of the target object 805 will besuperimposed on the direct view of the target object 805 on the surfaceof the half-silvered mirror 830. Because the jig 800 keeps the geometricorientations between the target object 805, mirror 830, imageacquisition device 810, and display 835 such that the desiredrelationships described above are satisfied, the jig 800 can be used tomake real-time use of the image acquisition device 810 unnecessary forcertain applications. The captured tomographic image 820 will not be inreal-time, but may still be useful if the target object is keptmotionless (or nearly motionless) during a procedure.

As described above, the present invention is preferably calibratedbefore use to ensure proper orientation of the components and display ofthe captured image. Specifically, the flat panel display must beoriented with respect to the half-silvered mirror such that thereflected tomographic image is correctly superimposed on the direct viewof the target object through the half-silvered mirror. This includescalibrating with both a geometric transform to ensure that the slice anddisplay are coplanar and an affine (or other) transform to scale theultrasound slice to its correct size, to adjust its aspect ratio, and tocorrect for skewing. Improved methods for both types of calibration arenow given.

Certain computers (e.g., SILICON GRAPHICS O2) are capable of mapping avideo image through an affine transform (or other transforms) inreal-time. The affine transform permits the tomographic slice displayedon the monitor to be translated, rotated, and anisotropically scaled.The calibration process becomes a matter of finding the optimalparameters for the affine transform.

Mapping location (x,y) to (x′,y′) with an affine transform isaccomplished by multiplying the homogeneous form of (x,y) by a 3×3matrix A. As shown in equation (1), an affine transform is capable ofmapping any triangle to any other triangle. If the mapping for 3locations is known, the computer can solve for the matrix A. Sincematrix A has only six unknown elements, it will have an explicitsolution (assuming the 3 locations are not collinear).

$\begin{matrix}{\begin{matrix}{\begin{matrix}x_{1}^{\prime} & x_{2}^{\prime} & x_{3}^{\prime}\end{matrix}} \\{\begin{matrix}y_{1}^{\prime} & y_{2}^{\prime} & y_{3}^{\prime}\end{matrix}} \\{\begin{matrix}{1\mspace{14mu}} & 1 & {\mspace{14mu} 1}\end{matrix}}\end{matrix} = {\begin{matrix}{\begin{matrix}a_{1,1} & a_{1,2} & a_{1,3}\end{matrix}} \\{\begin{matrix}a_{2,1} & a_{2,2} & a_{2,3}\end{matrix}} \\{\begin{matrix}{0\mspace{34mu}} & 0 & {\mspace{40mu} 1}\end{matrix}}\end{matrix}\begin{matrix}{\begin{matrix}x_{1} & x_{2} & {\; x_{3}}\end{matrix}} \\{\begin{matrix}y_{1} & y_{2} & y_{3}\end{matrix}} \\{\begin{matrix}{1\mspace{14mu}} & 1 & {\mspace{20mu} 1}\end{matrix}}\end{matrix}}} & (1)\end{matrix}$

Based on this theory, a “bead phantom” may be constructed with three (ormore) training beads suspended in roughly an equilateral trianglegenerally in the plane of the tomographic slice being taken. Anuncalibrated tomographic slice image from this phantom (which may besuspended in a water tank) will be captured and displayed on the flatpanel display. The water (or other fluid) will then be drained to avoiddiffraction errors at the air-water interface during visual inspection.The affine transform may then be found that maps the locations of eachof the three training beads in the stored ultrasound image to itscorresponding visual location (as seen through the half-silveredmirror). Assuming that the beads in the phantom are actually in theplane of the slice, the calibration process will be independent ofviewer location.

FIG. 9 shows one exemplary embodiment of an improved calibration methodfor the present invention which may be used to verify the abovecalibration method.

FIG. 9A shows a calibration target object known as a phantom 900. Thephantom 900 contains three (or more) targets (A,B,C) inside its surface.In FIG. 9A, the phantom 900 is shown with three beads A,B,C suspended bya thread which is not shown for clarity. The phantom 900 is subsequentlyfilled with a gel or other fluid (to enable tomographic imaging). Threetubes A*,B*,C* are mounted on the exterior surface of the phantom 900such that each is pointed at its corresponding interior bead (i.e., whenlooking through the tubes A*,B*,C*, the corresponding target bead A,B,Cwill be seen). The tubes may be placed after the target beads are inplace, or the beads (or other targets) may be inserted using thin rodsthrough the tubes as a guide for insertion (see, FIG. 9B).

In either case, after the tubes A*,B*,C* and target beads A,B,C arefixed into position, a small LED, light source, or other visualindicator is preferably inserted into the base (phantom side) of eachtube A*,B*,C* (see, FIG. 9C). These LEDs can be seen by the viewerlooking through the half-silvered mirror when the viewer is lookingparallel to the long axis of the tube. After the gel is inserted in thephantom and the LEDs are in place, the viewer can determine that he orshe is looking directly at a target bead A,B,C by seeing thecorresponding LED through its tube A*,B*,C*.

The slice through the three target beads A,B,C displayed on the flatpanel display 910 as A′,B′,C′ is reflected on the face of thehalf-silvered mirror 915. The corresponding reflections of the displayedimages of each target bead A′,B′,C′ can be aligned with the appropriateLED (in direct view) by using three independent (x,y) translations tocalculate an affine transform as described above. The co-planarity ofthe display's reflection and the ultrasound slice will have to bepre-established (through conventional techniques or as described below),and the three target beads will have to be visible in the ultrasoundslice. Given all of the above, a virtual target can then be displayed, aneedle (or other tool) may be inserted, and the geometric error can thenbe determined by measuring the intersection of the needle in theultrasound slice relative to the displayed virtual target (anywhere inthe image assuming linearity).

In addition to the affine transform described above, the components ofthe system must also be geometrically calibrated. Specifically thetomographic slice being captured must be coplanar with the flat paneldisplay being reflected onto the half-silvered mirror in order for thesimple geometric relationships of the present invention to be satisfied.The present method replaces an ad hoc “by eye” approach with a method inwhich targets within the slice and the virtual image are opticallydetermined to be coplanar using a video camera. The method may beundertaken either by the plane of focus of the video camera or by thecamera's exhibition of zero parallax when it is moved to a differentlocation.

For the focal plane approach, the water (if any) would be drained from atank holding the target to permit direct viewing of the target (such asbeads on a string).

Initially, the target beads and virtual image are viewed through a videocamera. The plane of focus of the camera would then be changed in somemeasurable way. If the target bead were coplanar with the virtual imageon the display, the video images of both would go in and out of focussimultaneously. Analysis of the frequency of the spectra of the videoimage sequence would reveal a single frame with a maximum highfrequency. Two such frames, however, would be observed if co-planarityhad not been achieved. After adjustments, this geometric calibrationconfirmation could then be repeated.

An alternate method to ensure co-planarity between the display and thecaptured slice may be based on parallax. This method preferably alignstargets in the tomographic image with the corresponding features in thevirtual image from one point of view, and then move the video camera toanother point of view to verify continued alignment. Errors thusintroduced in the alignment could be used to calculate corrections inthe position of the flat panel monitor, bringing its virtual image intoco-planarity with the ultrasound slice.

The parallax method could be used on the “virtual line of sight” phantom(with tubes containing LED's, FIG. 9) by providing multiple tubes foreach target pointing in different directions. For example, a system ofthree targets with six tubes may be used as an exemplary embodiment,with each set of three tubes converging on a different viewing location.Video cameras at these two locations would capture the required parallaxinformation.

The present invention may also take advantage of methods and apparatusesfor the guiding of invasive devices such as guiding the insertion of aneedle 1000 during a biopsy procedure. One preferred embodiment, shownin FIG. 10, introduces the use of an oriented light source 1005 tospecify a path to an internal target A. For example, in addition to theflat panel display 1010 (or in place of the flat panel display), a tubecontaining a light source 1015 (or simply a directional light sourcesuch as a laser) is positioned on the same side of the mirror as theviewer such that the virtual image of the light 1020 coincides with thedesired needle path 1000 within the patient. The combined view of thetwo image paths is depicted as image line 1025. The intended needle path1000 may be determined in a number of ways, including the use of imagedata from a scanner physically attached to the device or otherwisegeometrically related to the device by use of a jig if image data waspreviously acquired.

The laser or other light may also be reflected off of the flat paneldisplay and up to the half-silvered mirror. With this orientation, thelight source 1015 would not interfere with the reflection of thedisplayed tomographic slice, and the user's view 1025 may be lessobstructed. An almost limitless configuration of light sources andmirrors could be used to effect this reflection.

To determine light source placement, human and/or computer analysis ofthe image data (which may be 3D in the case of CT, MRI, 3D ultrasound,etc.) may be used to initially and/or continually provide a needle paththat hits the target while avoiding critical structures such asarteries. Light emanates from the tube such that the operator may findthe appropriate entrance location and initial orientation for theneedle. During insertion, continual guidance may be intuitively appliedby keeping the exposed shaft of the needle aligned with the virtualimage of the tube. Marks on the needle shaft can be visually alignedwith corresponding marks on the tube to determine depth of insertion andalert the operator when the target has been reached. FIG. 10 shows a 2Dversion of this embodiment.

The reverse of this “light” system could also be used to guide needlesor other tools into the target object. For example, a biopsy needle (orother tool) with small lasers mounted on the back of, and on oppositesides of, the needle which point parallel to the long axis of the needle(down its shaft) may be used. The lasers will be reflected off of thehalf silvered-mirror and down to the face of the flat panel display(with the half-silvered mirror oriented parallel to the plane of thedisplay and the image capture device). These lasers will appear as two“dots” (or other shapes) on either side of the target on the flat paneldisplay. These marking dots represent the position at which the needleis currently aimed in the target image. By watching the movement of thedots on the face of the display (or in the reflection inside thepatient), the needle or other tool may be properly guided into thetarget image.

One potential drawback of at least one embodiment of the presentinvention is the angle of refraction that occurs as the various imagespass through the glass of the half-silvered mirror. Because thereflected image bounces off the face of the mirror and the direct viewof the target object passes through the glass of the mirror, there willbe a refractive offset (error) between the two images due to thethickness of the glass.

FIG. 11 shows one exemplary apparatus for addressing this potentiallimitation. In FIG. 11, a cross-section of the mirrored surface 1100 ofthe half-silvered mirror is shown between two equal thickness pieces ofglass 1110, 1120 (with the same refractive index). With this structure,both the direct view and the reflected image will pass through the glass1110, 1120 and be refracted an equivalent amount. In other words, thesame error due to refraction will be imparted to each of the imagelines. This orientation will therefore preferably correct for theproblems of refraction in the half-silvered mirror.

The half-silvered mirror embodiments of the present invention (describedabove) could also be altered to use a Holographic Optical Element (HOE)instead of the mirror. This is just another example of the generalized“partially reflective, partially transparent, surface.” For example,FIG. 12 shows one presently preferred embodiment of the presentinvention including an ultrasound transducer 1205 with an HOE or“holographic plate” 1230 and a micro-mirror MEMS chip 1210 mountedthereon. An HOE 1230 is the hologram of an optical system that servesthe same function normally performed by mirrors and lenses. Themicro-mirror MEMS chip 1210 preferably contains a large array ofindividual micro-mirrors, for example in a grid pattern as shown in FIG.12.

This system works by reflecting laser light 1220 off of each of themicro-mirrors that make up the micro-mirror device 1210. The capturedtomographic slice image is used to control (via circuitry 1250) each ofthe tiny micro-mirrors 1210 to determine whether or not the laser light1220 is reflected up the HOE 1230. Specifically, the output of a laser1220 may be routed via a fiber optic cable 1225 to the micro-mirror chip1210 so that coherent light may be reflected onto the HOE 1230 by anygiven micro-mirror when that particular micro-mirror is activated. Thisactivation is determined by a given pixel in a video signal from theultrasound transducer which represents a voxel (A) in the ultrasoundslice.

The video image from the image capture device 1205 (e.g., ultrasoundscanner) may be appropriately scaled, rotated, translated and cropped(via circuitry 1250) to occupy the array of micro-mirrors on the chip1210. The HOE 1230 is preferably designed so that light reflected by agiven micro-mirror appears to emanate from a particular location withinthe target, namely, the location of the voxel (A) in the tomographicscan corresponding to that particular pixel in the ultrasound image.Thus, the ultrasound slice as a whole appears to emanate in real-timefrom within the target object 1200 at its actual location.

Although FIG. 12 shows the major components of such a holographicsystem, parts of the optical system (mirrors and/or lenses) required todirect the laser light 1220 onto the micro-mirror chip 1210 and thephase modulation system used to reduce speckle normally resulting fromcoherent light are not shown for clarity. These and other desiredcomponents of a laser light system are well known to one skilled in theart. Additionally, FIG. 12 depicts only the particular case involving anultrasound scanner capturing a single tomographic slice. Othertomographic imaging modalities, including those that operate in 3D(described above), may also be displayed using this method, since the 2Darray of micro-mirrors can be mapped into a 3D space within the patient.

The holographic system establishes the geometric relationship betweenthe micro-mirrors and the virtual tomographic image. The content of theimage is established by the video signal from the ultrasound scanner,whose pixels control to the individual elements of the micro-mirrordevice. It is further noted that the HOE embodies a transform functionfor creating a magnified view-independent virtual image that could becreated using lenses and mirrors, whereas non holographic opticalelements would be very large and somewhat less desirable.

Difficulties may arise with the use of the micro-mirror device due tothe small angle of deflection capable in the micro-mirrors. Therefore,other devices could be used in place of the MEMS device such as a LiquidCrystal Display (LCD) or any device capable of switching on/offindividual localized sources of monochromatic light directed at the HOE1230. For example, FIG. 13 shows an exemplary LCD shutter with laserlight (attempting) to pass through the shutter towards the HOE. In FIG.13, the directional laser light passes through the non-polarizingapodiser and diffuser. The LCD shutter and the polarization layer workin tandem to determine the amount of light that passes through each LCDshutter based on the relative polarization of these two layers.

Each LCD shutter can be quickly turned on/off so as to direct theultrasound image up to the HOE. In much the same was as with the MEMSdevice, each individual LCD shutter may correspond to a pixel in thecaptured tomographic slice and will be turned on/off based on whetherthat pixel currently exists in the image. With sufficiently fastswitching, the same result on the face of the HOE is achieved.

A simplified version of the HOE could also be used in the presentinvention as depicted in FIG. 14. For example, it is known in generaloptics that the hologram of an ideal mirror is simply a grating. Thehologram of a simple convex lens is a zone-plate, i.e., a series ofconcentric rings that create an interference pattern.

The hologram pattern can be calculated using the lens equation:1/x+1/y=1/f (where x=distance from lens on one side; y=distance fromlens on the other side; and f=focal length). A series of correspondencepoints thus exists on opposite sides of the lens which can be used toproject the array of emission points (from the MEMS micro mirrors or LCDdevice) to the corresponding virtual locations in the ultrasound slice.In other words, a single hologram can map the whole image, rather than aseparate hologram for each pixel. The same approach could be used to mapfrom the MEMS (or LCD) device mounted parallel to the HOE to a “C-mode”slice from a 3D ultrasound machine, as described earlier and shown inFIG. 3.

The hologram of a tomographic slice, therefore, would generally be anoff-axis sector of this zone plate as shown in FIG. 14. The partialspherical etches in the HOE will produce spherical wavefronts fromA′,B′,C′ when illuminated at A,B,C with coherent light, because the zoneplate acts as a lens. FIG. 14B shows a side view of the device. However,the hologram will have some “keystone-shaped” distortions as shown inFIG. 14. These distortions can be compensated for by a correspondingdistortion of the image sent to the MEMS or LCD device (from circuitry1250). There may also be problems with resolution in parts of the imagedue to inefficient use of the entire MEMS or LCD device. The 3Dultrasound version might also experience distortion or curvature in the“C-mode” slice, and these may be corrected by judicious selection of theparticular voxels to be displayed.

The resulting HOE is a much simpler 2D HOE that could be implemented asa “surface plate” rather than the HOE described above that would likelyrequire a “volume plate”. The surface plate is easier to manufacture andmay be much cheaper to produce. The present HOE may be manufactured byetching the calculated pattern into a chromium coated glass plate.

The present invention may also be used to display multiple tomographicslices simultaneously. For example, this can be accomplished using themirror orientation described in several embodiments above with theaddition of multiple flat panel displays. If each of the flat paneldisplays was oriented with respect to the viewer-side face of thehalf-silvered mirror (according to the calibration methodologiesdescribed above), then multiple tomographic slices could be superimposedon the direct view of the target object on the face of the half-silveredmirror. This may give the impression of seeing an internal 3D view ofthe target object. However, because all of the various images arereflected to a flat surface, the resulting combined image may bedifficult to comprehend in certain circumstances.

Alternatively, multiple MEMS micro-mirror devices or multiple LCDshutters could be used to display multiple captured slice imagessimultaneously in real-time on the HOE. In the same way as multiple flatpanel displays may be used with the half-silvered mirror embodiments,two or more micro-mirror devices or LCD shutters could be oriented suchthat they direct laser light to the HOE to simultaneously show multipletomographic slice images on the HOE. Alternatively, by dividing a singleMEMS into several sections, each with a different holographic transform,multiple slices could appear to be displayed on the HOE simultaneously.However, cross-talk between the slices may be easier to minimize if eachimage is displayed on its own MEMS chip, rather than sharing sections ofa single MEMS chip. The HOE device also has the advantage over themirror that a slice displayed roughly “edge-on” could be viewed fromeither side by moving the viewpoint. This multiple-device orientationmay reduce alignment problems related to showing multiple slices on onedevice, but it may also increase the cost of the system. Therefore,different orientations of these components will be preferred fordifferent applications.

In other presently preferred embodiments, the apparatuses of the presentinvention have been adapted to allow a remote expert to participate inthe evaluation of an image or the manipulation of the subject that isbeing imaged. In such embodiments, the system is preferably divided intotwo portions. The first portion of the system is located local to thesubject being scanned (“the local site”), while the second portion islocated distal to the subject being scanned (“the remote site”). Thefirst portion preferably employs real-time tomographic reflectiondevices as described hereinabove preferably modified so that they areable to transmit the tomographic or superimposed image from the localsite to the remote site.

At the remote site, the tomographic or superimposed image may bedisplayed on a screen so that it may be observed by a trained expert.The trained expert may then provide input based on the image to thelocal user through a communication channel. The trained expert may, forexample, interact with the image via an electronic interface to identifyobjects of interest within the image. The trained expert may place oneor more electronic markers (e.g., arrows, cursors, writing, shapes, orhighlighting) on the tomographic image. The location and nature of thoseelectronic markers may then be transmitted back to the local site. Basedon the information regarding the location and nature of the electronicmarkers, the electronic markers are then preferably displayed at theappropriate location on the captured image. In these embodiments, thecombined image of the captured image and electronic marker is displayedso that the combined image is overlaid onto the direct view of thetarget object, as described fully hereinabove. The electronic markersmay be employed by users at the local site to manipulate the subjectappropriately, as described hereinbelow.

In other embodiments, after the trained expert at the remote locationplaces an electronic marker on the captured image, both the capturedimage and the electronic marker are transmitted from the remote locationto the local site. While such a transmission may involve transmission ofhigh volumes of data between the two sites, such configurations of thepresent system may be preferable depending on the particularcircumstances that are encountered.

In still other embodiments, the trained expert by provide audio feedbackto the local user. In those embodiments, the image may be transmitted tothe remote site where the trained expert may evaluate it as describedabove. Audio feedback may be transmitted back to the local user toprovide interpretation of the image or instruction to the local userthrough the use of a microphone or headset. In such instances, the imagemay or may not be transmitted back to the local user from the remotesite.

The operation of these preferred embodiments of the present inventionmay be better understood through the following example. The localportion of an assembly of the present invention 1500 is show in FIG. 15.FIG. 15 displays a two dimensional ultrasound transducer 1512 fixedlyattached to a rigid frame 1518. The transducer 1512 is preferably aconventional ultrasound transducer that captures a tomographic slice1514 of the interior portion of the target object 1516. In thisinstance, the target object 1516 is a human patient whose vein 1517 alocal user desires to access.

Extending vertically from the middle region of the rigid frame 1518 is ahalf-silvered mirror or other semi-transparent, semi-reflective material1520. The half-silvered mirror 1520 allows a user 1522 to look throughthe mirror 1520 at a target object 1516 located on the other side of themirror 1520 at the same time that a second image 1534 reaches the mirrorvia path 1524 and is reflected by the front surface of the mirror (at1528). In this manner, the direct target object image seen along path1526 and the tomographic image along path 1524 can be combined alongpath 1530 in the field of view of the user 1522.

At the opposite end of the rigid frame 1518 from the transducer 1512 isa flat panel display 1532 showing the image or the ultrasound data (orother tomographic modality) 1534, with the image portion 1534 facingupwards, as further described below. Just as described above, when auser 1522 looks at a target object 1516 through the half-silvered mirror1520, the ultrasound display image 1534 will be reflected along line1524 onto the front face of the half-silvered mirror (at 1528). The user1522 thus observes a combination or superimposition of the direct targetobject image along line 1526 and the reflection of the ultrasound imagefrom line 1524.

The presently preferred embodiments of the present invention preferablyinclude a transmitter/receiver 1540 which is adapted to transmit theimage from the tomographic scanner 1512 to a remote location. One ofskill in the art will recognize that numerous transmission protocols(e.g., BLUETOOTH, other radio frequency transmission, infrared, or insome cases even hard-wired connections such as a fiber optic cable) maybe employed to transmit the image information to the remote location.

FIG. 16 displays the assembly of components 1600 of the presentinvention preferably found at the remote location. The remote assemblypreferably includes a transmitter/receiver 1604 that is adapted toreceive the transmitted tomographic image. The transmitter/receiver 1604is preferably connected to a screen 1608 onto which the tomographicimage 1612 may be displayed. A remote user 1616 may thereby view thetomographic image 1612 on the screen 1608. The remote user 1616 ispreferably provided with an electronic interface 1620 by which is ableto interact with the tomographic image 1612. The electronic interface1620 may be a keyboard, joystick, or mouse as shown in FIG. 16. Theelectronic interface 1620 allows the remote user 1616 to place and movean electronic marker 1624 (here, an arrowhead) on the image 1612. Theelectronic marker 1624 may be used to identify a target within thesubject, such as a vein 1517 as shown in FIG. 16. The tomographic image1612 with electronic marker 1624 is then preferably transmitted back tothe local site (FIG. 15) via transmitter/receiver pair 1604 and 1540. Incertain presently preferred embodiments, the electronic marker 1624 maybe manipulated by the remote user and transmitted to the local user inreal time, and thus provide extremely accurate guidance to the localuser.

At the local site (FIG. 15), the tomographic image and electronic markerwould be displayed on screen 1532 and thus displayed to the local user1522. The local user 1522 could use the displayed electronic marker 1624to aid in performing the assigned task on the object—in this instanceallowing access to vein 1517. The local user 1522 would insert thecatheter into the vein 1517 at the location indicated by the electronicmarker 1624. The present invention may also include a bidirectionalaudio channel so that the local user may report conditions to andreceive instructions from the remote user regarding numerous localfactors, including the optimal placement of the imaging device. Thus,the present invention allows a remote user to communicate his or herexpertise to a local user so that the task may be performed on thetarget object appropriately.

FIG. 17 displays another presently preferred embodiment of the presentinvention that includes a camera 1735 that is adapted to capture theimage from the mirror at 1528 electronically. In some embodiments,camera 1735 may be placed onto the helmet 1737 of a local user 1522 sothat there is correspondence between the view of the local user 1522 andthe camera 1735. Regardless of the exact location of the camera themerger of the tomographic image and the direct view of the patient isviewpoint independent, as described above, so that the location of thecamera may be varied widely depending on the circumstance with no lossof information. In certain embodiments, the superimposed image capturedby the camera 1735, as well as the tomographic image, may be transmittedvia the transmitter/receiver 1540. At the remote location, both thetomographic and superimposed images may be displayed on a screen. Theremote user may manipulate each of the images independently and theinformation may be transmitted by a transmitter/receiver (see FIG. 16)back to the local site.

The remote user 1616 may also use a computer to manipulate the image.For example, the remote user 1616 may employ image analysis software toenhance the image prior to sending back to the local site. The remoteuser may also manipulate the image to provide color-based instruction tothe local user. Numerous manual, semi-automated, and automated imagemanipulations are well known to those of ordinary skill in the art andmay be employed within the context of the present invention. As anotherexample, the remote user may overlay an instructive image onto thecaptured image to aid the local user in performing the task. Theinstructive image may be an image that was previously obtained from thesubject or represent prior knowledge of normal anatomy. Alternatively,the image may be enhanced through the localization of anatomicalstructures that may not be clearly visible in the original capturedimage. Each of those alterations could be applied at the remote siteusing a remote computer by the remote user. The improved and alteredimage could thereby provide the local user with an enriched image sourcethat might aid in performing any procedure at the local site.

These preferred embodiments of the present invention are of particularutility where the expertise of a highly trained individual would beimportant and/or required for performing a task. For example, in medicalprocedures the perspective and training of a clinician may be imperativefor appropriately performing a medical procedure. In some situations,such as on a battlefield, a clinician may be unable to directly attendto an injured individual. A medic or solider without any medicaltraining may be forced to insert a catheter into a vein of a woundedsoldier to provide needed plasma. The imaging devices and systems of thepresent invention may allow a remotely located clinician to interpret anultrasound image appropriately for a locally placed medic. The systemsand apparatuses of the present invention may also be used in thedecommissioning of military ordinances. For example, a SCUBA diver mayemploy sonar-based real time tomographic imaging as described above toimage the interior of a mine. The diver could obtain guidance from aremote expert in underwater mines who could point the diver to theappropriate wires to clip within the mine to allow decommissioning ofthe weapon.

Nothing in the above description is meant to limit the present inventionto any specific materials, geometry, or orientation of elements. Manypart/orientation substitutions are contemplated within the scope of thepresent invention and will be apparent to those skilled in the art. Theembodiments described herein were presented by way of example only andshould not be used to limit the scope of the invention.

1. An imaging system, comprising: a local assembly proximal to a targetobject to be imaged, comprising: an image capture device for capturingan image of the internal structure of the target object; a first displayfor displaying a captured image from the image capture device; apartially reflective surface oriented to reflect the captured image toan operator of the imaging device, such that the reflected capturedimage is merged with a direct view of the target object independent ofthe viewing location of the operator when the angle between the surfaceand the display is equal to angle between the surface and the imageplane; and a first transmitter-receiver adapted to transmit saidcaptured image to a second transmitter-receiver; and a remote assemblydistal from said target object, comprising: said secondtransmitter-receiver adapted to receive said captured image; a seconddisplay for displaying the captured image; wherein said firsttransmitter-receiver and said second transmitter-receiver comprise acommunication channel adapted to transmit information from a user atsaid remote assembly to said local assembly.
 2. The system of claim 1,wherein said information is audio information, electronic information,or combinations thereof.
 3. The system of claim 2, wherein said secondassembly further comprises an electronic interface adapted to place anelectronic marker onto said captured image, further wherein said secondtransmitter-receiver is adapted to transmit information about thelocation and nature of said electronic marker to said firsttransmitter-receiver over said communication channel.
 4. The system ofclaim 3, wherein second transmitter-receiver is further adapted totransmit an altered captured image.
 5. The system of claim 1, whereinsaid image capture device is selected from the group consisting of:ultrasound-based probe, radar-based probe, LIDAR-based probe, andSONAR-based probe.
 6. The system of claim 1, wherein said first assemblyfurther comprises an electronic camera.
 7. The system of claim 6,wherein said electronic camera is adapted to capture said merger of saidreflected captured image and said direct view of the target object. 8.The system of claim 7, wherein said first transmitter-receiver isadapted to transmit both said captured image and said merger of saidreflected capture image and said direct view of the target object tosaid second transmitter-receiver.
 9. The system of claim 8, wherein saidsecond display is adapted to display both said captured image and saidmerger of said reflected captured image and said direct view of thetarget object.
 10. The system of claim 1, wherein said electronic markeris selected from the group consisting of cursor, arrow, writing, shapes,and highlighting.
 11. A method of imaging a target object, comprisingthe steps of: at a first location, capturing an image from said targetobject using an image capture device; transmitting the captured image toa second location via a first transmitter/receiver; at said secondlocation, receiving said transmitted captured image; displaying saidcaptured image; providing an individual at the second location with aninterface, wherein said interface allows said individual to communicateinformation about said captured image to said first location; at saidfirst location, receiving said information about said captured image;displaying said captured image on a electronic display; and reflectingsaid captured image to an operator, such that the reflected capturedimage is merged with a direct view of the target object independent ofthe viewing location of the operator when the angle between the surfaceand the display is equal to angle between the surface and the imageplane.
 12. The method of claim 11, wherein said information about saidcaptured image is audio information, electronic information, orcombinations thereof.
 13. Original) The method of claim 12, wherein saidinterface is an electronic interface that allows said individual toplace an electronic marker on said captured image.
 14. The method ofclaim 13, wherein said electronic information is information about thelocation and nature of said electronic marker.
 15. The method of claim14, wherein said step of displaying said captured image on a electronicdisplay further includes displaying said electronic marker on saidcaptured image.
 16. The method of claim 15, wherein said step ofreflecting said captured image further includes reflecting saidelectronic marker on said captured image.
 17. The method of claim 11,wherein said image capture device is selected from the group consistingof ultrasound-based probe, radar-based probe, LIDAR-based probe, andSONAR-based probe.
 18. The method of claim 11, wherein said firstassembly further comprises an electronic camera.
 19. The method of claim18, wherein said electronic camera captures said merger of saidreflected captured image and said direct view of the target object. 20.The method of claim 19, wherein said first transmitter/receiver isadapted to transmit both said captured image and said merger of saidreflected capture image and said direct view of the target object tosaid second transmitter/receiver.